System and methods for interacting with a control environment

ABSTRACT

A system and methods for facilitation of user interactions with an electronic device. A number of user interface methods are described and may be used alone or in combination with one another to present an enhanced interface to a user. A method of providing user interaction using a compact status indicator is described. A method for providing a virtual scroll wheel to a user for interaction with content sets is described. A method for allowing a user to dynamically modify a scalable user interface is described. A method for providing gesture based input to a user via a virtual gesture pad is described. A method of providing an interactive graphic search query interface is described. A method for indicating and selecting available content type is described.

RELATED APPLICATION INFORMATION

This application claims the benefit of and is a Divisional of U.S.application Ser. No. 11/025,216.

Through U.S. application Ser. No. 11/025,216, this application claimsthe benefit of U.S. Provisional Application Ser. No. 60/534,608 filed onJan. 6, 2004 and, as a continuation-in-part application, the benefit ofU.S. Published Patent Applications Nos. 2003/0103088 filed on Nov. 6,2002, 2003/0117427 filed on Nov. 4, 2002, and 2002/0143805 filed Jul.13, 2001.

All of these prior applications are incorporated herein by reference intheir entirety.

BACKGROUND OF THE INVENTION

This invention relates generally to user interfaces for electronicdevices. Exemplary devices include personal digital assistants (“PDAs”),Web Tablets, touch screen remote controls, mobile phones, lap-topcomputers, and the like.

SUMMARY OF THE INVENTION

In accordance with the description that follows, a system and method isprovided for enabling enhanced user interaction, information display,and interface selection for electronic devices having graphic displayand touch screen capabilities. An understanding of the objects,advantages, features, properties and relationships of the invention willbe obtained from the following detailed description and accompanyingdrawings which set forth illustrative embodiments and which areindicative of the various ways in which the principles of the inventionmay be employed.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various aspects of the invention,reference may be had to preferred embodiments shown in the attacheddrawings in which:

FIG. 1 illustrates exemplary devices for which the methods of thepresent invention may be implemented;

FIG. 2 illustrates an exemplary method for browsing and indicatingcontent status using an exemplary prior art scroll-bar;

FIGS. 3-9 illustrate exemplary methods for browsing and indicatingcontent status using an illustrated, exemplary compact status indicator;

FIGS. 10 and 11 illustrate an exemplary method for scrolling throughcontent using an illustrated, exemplary virtual scroll wheel;

FIGS. 12-17 illustrate exemplary methods for presenting a graphic userinterface having an illustrated, exemplary scalable interface;

FIGS. 18-22 illustrate an exemplary method for gesture based softwareapplication control using an illustrated, exemplary virtual gesture padinterface;

FIGS. 23-26 illustrate an exemplary data display and selection methodusing an illustrated, exemplary graphical search query interface;

FIG. 27 illustrates an exemplary graphic user interface method fordisplaying and selecting content types supported by media servers orrenderers; and

FIGS. 28-30 illustrate exemplary methods for presenting a graphic userinterface with exemplary graphical indications of device mode and state.

DETAILED DESCRIPTION

A universal remote control application and associated interfacingmethods are provided for executing on a portable electronic device 10.By way of example, representative platforms for the device 10 include,but are not limited to, devices such as remote controls, personalcomputers, lap-top computers, Smart Displays, Web Tablets and/or PDAsmanufactured by HP/Compaq (such as the iPAQ brand PDA), Palm, Visor,Sony, etc., mobile phones (i.e., Microsoft based Smart Phones, Palm OSand/or Symbian OS based phones), personal gaming systems (i.e., NintendoGameBoy, Nokia N-Gage), etc. Thus, a preferred underlying platformincludes a processor coupled to a memory system comprising a combinationof ROM memory, non-volatile read/write memory, and RAM memory (a memorysystem); a key matrix in the form of physical buttons; an internal clockand timer; a transmission circuit; a power supply; a touch screendisplay screen to provide visible feedback to and accept input from auser (i.e., via virtual buttons or keys); and I/O circuitry for allowingthe device to exchange communications with an external computer such asserver and/or client and consumer appliances. Additional input/outputcircuitry, such as a IR transmitter(s) and/or receiver(s), microphone,loudspeaker or earphone jack, barcode or RFID reader, etc., may also beprovided.

To control the operation of the device 10, the memory system includesstored programming instructions that are intended to be executed by theprocessor. In this manner, the processor may be programmed to controlthe various electronic components within the device 10, e.g., to monitorpower, to cause the transmission of signals, etc. Within the memorysystem, the ROM portion of memory is preferably used to store fixedprogramming and data that remains unchanged for the life of the product.The non-volatile read/write memory, which may be FLASH, EEPROM,battery-backed up RAM, “Smart Card,” memory stick, or the like, ispreferably provided to store user entered setup data and parameters,downloaded data, etc., as necessary. RAM memory may be used by theprocessor for working storage as well as to hold data items which, byvirtue of being backed up or duplicated on an external computer (forexample, a client device) are not required to survive loss of batterypower. While the described memory system comprises all three classes ofmemory, it will be appreciated that, in general, the memory system canbe comprised of any type of computer-readable media, such as ROM, RAM,SRAM, FLASH, EEPROM, or the like alone or in various combinations.Preferably, however, at least part of the memory system should benon-volatile or battery backed such that basic setup parameters andoperating features will survive loss of battery power. In addition, suchmemories may take the form of a chip, a hard disk, a magnetic disk,and/or an optical disk without limitation.

For commanding the operation of appliances of different makes, models,and types, the memory system may also include a command code library.The command code library is comprised of a plurality of command codesthat may be transmitted from the device 10 under the direction of thestored programming instructions for the purpose of controlling theoperation of an appliance. The memory system may also includeinstructions which the processor uses in connection with thetransmission circuit to cause the command codes to be transmitted in aformat recognized by an identified appliance. While an exemplarytransmission circuit preferably utilizes infrared transmissions, it willbe appreciated that other forms of wired or wireless transmissions, suchas radio frequency, may also be used.

To identify appliances by type and make (and sometimes model) such thatapplication(s) of the device 10 are adapted to cause the transmission ofcommand codes in the format appropriate for such identified appliances,information may be entered into the device 10. Since methods for settingup an application to cause the transmissions of commands to control theoperation of specific appliances are well-known, they will not bedescribed in greater detail herein. Nevertheless, for additional detailspertaining to remote control setup, the reader may turn to U.S. Pat.Nos. 6,225,938, 6,157,319, 4,623,887, 5,872,562, 5,614,906, 4,959,810,4,774,511, and 4,703,359 which are incorporated herein by reference intheir entirety.

To cause the device 10 to perform an action, the device 10 is adapted tobe responsive to events, such as a sensed user interaction with one ormore keys on the key matrix, a sensed user interaction with the touchscreen display, a sensed user voice/sound input, a sensed gesture, asensed movement, or other non-traditional command input methods, or asensed signal from an external source such as a remote computer. Inresponse to an event, appropriate instructions within the memory systemare executed. For example, when a hard or soft command key associatedwith a remote control application is activated on the device 10, thedevice 10 may read the command code corresponding to the activatedcommand key from the memory system and transmit the command code to anappliance in a format recognizable by the appliance. It will beappreciated that the instructions within the memory system can be usednot only to cause the transmission of command codes to appliances butalso to perform local operations. While not limiting, local operationsthat may be performed by the device that are related to the remotecontrol functionality include favorite channel setup, macro buttonsetup, command function key relocation, etc. Examples of such localoperations can be found in U.S. Pat. Nos. 5,481,256, 5,959,751,6,014,092, which are incorporated herein by reference in their entirety.

As discussed, the underlying platform of the device 10 preferablycomprises a general purpose, processor system which is controllable bystored programming instructions, i.e., software. The software mayinclude routines, programs, objects, components, and/or data structuresthat perform particular tasks that can be viewed as an operating systemtogether with one or more applications. The operating system, such asthe “Windows CE” brand operating system or the like, provides anunderlying set of management and control functions, device drivers, andthe like which are utilized by the various applications to offer theuser functions such as a calendar, address book, spreadsheet, notepad,Internet browsing, etc., as well as control of appliances. Thus, it isto be understood that applications in addition to or complimentary withthe remote-control-like application can also be supported by the device10 and, as such, in terms of the internal software architecture, theremote-control-like application may be but one of several possibleapplications which may co-exist within the device 10.

In terms of providing operating system functionality, it should also beunderstood that the demarcation between the device 10 and a host/clientcomputer, described in greater detail hereinafter, may vary considerablyfrom product to product. For example, at one extreme the device 10 maybe nothing more than a slave display and input device in wirelesscommunication with a computer that performs all computational functions.At the other extreme, the device 10 may be a fully-functional computersystem in its own right complete with local mass storage. It is also tobe appreciated that a hardware platform similar to that described abovemay be used in conjunction with a scaled-down operating system toprovide remote control functionality only, i.e., as a standaloneapplication. In all cases, however, the principles expressed hereinremain the same.

To provide a means by which a user can interact with the device 10, thedevice 10 is preferably provided with software that implements agraphical user interface. The graphical user interface software may alsoprovide access to additional software, such as a browser application,that is used to display information that may be received from anexternal computer. Such a graphical user interface system is describedin pending U.S. application Ser. Nos. 09/905,396, 60/334,774, and60/344,020 all of which are incorporated herein by reference in theirentirety. Though described in the below embodiments in conjunction withremote control software applications, it will be understood andappreciated that the various user interface and interaction basedfeatures described herein may be used in conjunction with any softwareprogram or application and are thus not specifically limited toapplications directed to control of consumer appliances.

Compact Status Indicator

For maximizing available display area on a user interface and forsimplifying the process of indicating content status and/or browsingthrough a large content set, the device 10 utilizes a compact statusindicator interface 11. In particular, the compact status indicatorinterface 11 is designed to overcome both size and functionalityrestraints of portable electronic devices, and present a full featured“scroll-bar” like interface and status indication to a user for browsingthrough a content set (i.e., a set of text or other displayed datareferencing files, photos, music, videos, program guide information,etc., that cannot be conveniently displayed within the available displayarea of a device). In particular, presently to navigate within largedata or content sets, the user must use a traditional scroll-bar typeinterface such as shown in FIG. 2. While traditional scroll-barinterfaces are able to present indicators to the user as to the size ofthe content set currently being browsed, and allow the user to browsethrough the content using one or more methods of interacting with thescroll-bar (e.g., via a mouse, cursor, stylus, etc.), such scroll-barsrequire a relatively large amount of space on the display and thus limitthe amount of content display/viewing area. As will be appreciated, theloss of available display area (for content viewing or any other desiredfunction) due to traditional scroll-bar type interfaces is especiallyproblematic when implemented on relatively small electronic deviceshaving correspondingly small displays. Additionally, for certain deviceswith restrictively small display screens such as mobile phones and thelike it may not be feasible to implement any traditional scroll-barstyle interface due to display space requirements, even when such aninterface would be useful to a user. The compact status indicator of thecurrent invention overcomes these limitations while presenting a fullfunction content interface to a user.

Looking now to FIGS. 3-6, compact status indicator 11 as depicted isgenerally circular in shape and includes at least two portions 110 and114 for indicating to a user the current status of an associated contentset. Generally, the entirety of compact status indicator 11 (portions110, 112, and 114 collectively) represent the entirety of the contentset currently able to be browsed by a user. Portion 110 represents theamount of content which has been browsed through, and portion 114represents the remaining amount of content available to be browsed by auser. Compact status indicator 11 may optionally include portion 112 forindicating to a user the relative amount of content the user iscurrently viewing within the content display area 12 as compared to thetotal content set being browsed. As such, portion 112 is configured (viaprogramming in or accessible to device 10) to dynamically changedepending on the size of content display area 12 relative the displayedcontent set, similar to the familiar operation of portion 112′ of thetraditional scroll-bar shown in FIG. 2.

Compact status indicator 11 may also indicate the relative location ofthe then displayed content set on a display via the position of portion112. For example, in 11 a the content set as then displayed in contentdisplay area 12 is substantially one-quarter of the way through theamount of available content to be displayed, in 11 b the content set asthen displayed in content display area 12 is substantially one-half ofthe way through the amount of available content to be displayed, and in11 c the content set as then displayed in content display area 12 issubstantially thee-quarters of the way through the amount of availablecontent to be displayed. In the event that portion 112 is notimplemented in a specific embodiment and/or has been turned off by theuser, it will be appreciated that a similar indication may be achievedby the relative position of the boundary between portions 110 and 114.

The inventive aspects of the present compact status indicator will beunderstood and appreciated by the foregoing description and associateddrawings, however since in one exemplary method compact status indicator11 is configured and operated to replace traditional scroll-bar 11′,reference to corresponding portions may be seen in FIGS. 2 and 3 forclarification. Specifically, portion 110′ of traditional status-bar 11′corresponds generally to portion 110 of compact status indicator 11.Portion 112′ of traditional scroll-bar 11′ corresponds generally toportion 112 of compact status indicator 11, which changes dynamicallybased on the proportion of displayed content to total content availablefor display from a given content set. Portion 114′ of traditionalscroll-bar 11′ corresponds generally to portion 114 of compact statusindicator 11.

FIGS. 4-9 illustrate exemplary methods for using the compact statusindicator of the present invention to interact with a content set orother element on a graphical user interface. A status indicator onlyembodiment is depicted in FIGS. 4-7, and generally serves only toindicate to a user the portion of content browsed though at a giventime. An interactive compact status indicator embodiment is shown inFIG. 8, and includes portion 1 12 for indicating to a user the portionof content currently displayed relative to the total content set. Theinteractive indicator of FIG. 8 is also operative in conjunction with amouse, stylus, or other similar user interactivity element to causecontent in content display area 12 to advance further through thecontent set (i.e., scroll). Such interaction is more fully illustratedin FIG. 9, wherein a stylus is shown being operated (e.g., dragged ormoved) in an arcuate fashion about an inner portion of the enlargedcompact status indicator 111. Portion 112 may be configured in thisembodiment as an interactivity guide (e.g., a portion outside of whichuser interactions with the touch screen will not cause movement of thecontent set) which moves radially about the compact status indicator asthe user drags portion 112 in a clockwise or counterclockwise fashionwith respect to the entirety of the compact status indicator. As will beappreciated from FIGS. 8 and 9, the compact status indicator may be usedfor interacting with a content set either in a compact form generallylocated on the periphery of a content display area (as shown in FIG. 8)or in an enlarged, superimposed interface directly above the content setto be controlled (as shown in FIG. 9). The enlarged status indicator 111can be configured in a semi-transparent fashion such that viewing of thecontent set lying below the enlarged status indicator (as illustrated inFIG. 9) will be uninhibited for the user.

It will be understood and appreciated that the size, placement on aparticular user interface or electronic device, shading, coloring, andother “look and feel” elements of the compact status indicator of thecurrent invention may vary widely without departing from the spirit andscope of the current invention. Additionally, the particular methods andtechniques for generating and allowing interaction with the compactstatus indicator will be apparent from the descriptions herein, as wellwithin the routine skill of a programmer skilled in the art. Forinstance, when implemented in conjunction with a remote controlapplication, the compact status indicator of the current invention canbe configured to both provide status of displayed content sets,interaction with such content sets, as well as accomplish other desireduser interactions with home appliances using the control methodsdescribed above. Thus, the compact status indicator may also be operablein a given device mode to not only indicate a relative position withinan appliance state but also to provide interactive control of thatstate, such as volume control (wherein the pie-shaped indicator has arange from no volume to maximum volume), channel control/surfing(wherein the pie-shaped indicator has a range from the lowest availablechannel to the highest available channel), etc. In connection withoperating the device 10 in this manner, interaction with the indicatormay transmit appropriate commands such as volume up/down, channelup/down, etc. In certain circumstances, an absolute value may also betransmitted—such as a specific channel number if the device 10 isprovided with a correspondence between a position of the indicator 112within the pie-graph and a channel number within the range of channelnumbers available.

Virtual Scroll Wheel

For further maximizing available display area on a user interface andfor simplifying the process of browsing through a large content set, adevice 10 may utilize a virtual scroll wheel interface 14. Inparticular, the virtual scroll wheel interface 14 is designed toovercome both size and functionality restraints of a small electronicdevice, and present a full featured “mechanical scroll-wheel” or“mechanical jog-dial” like interface to a user for browsing through alarge content set (i.e., a set of text or other displayed datareferencing files, photos, music, videos, etc. that cannot beconveniently displayed within the available display area of a device)using a finger, stylus, or other user interactivity element. Inparticular, presently to navigate within large data or content sets, theuser must use a traditional scroll-bar type interface such as shown inFIG. 2, and/or an external, mechanical scroll-wheel or jog-dial typeinterface as are well known in the art. While traditional scroll-bar,scroll-wheel, and jog-dial interfaces are able to allow the user tobrowse through the content using one or more methods of interacting withthe interface (e.g., via a mouse, finger, cursor, stylus, etc.), suchscroll-bars require a relatively large amount of space on the displayand thus limit the amount of content display/viewing area. Specificallyin the case of scroll-bars, finger based interaction (i.e., using onesthumb as would be possible using a physical scroll-wheel interface)becomes difficult to implement as scroll-bars were designed forinteraction via cursor or stylus type interactivity elements, and becomeunwieldy when operated with large blunt objects such as fingers.Likewise, physical scroll-wheels and jog-dial are both costly toimplement on small electronic devices, and consume valuable space on thecase of the device. As will be appreciated, the difficulty in browsingcontent via a scroll-bar using ones thumb, and the loss of availabledisplay and/or device case area (for content viewing or any otherdesired function) due to traditional scroll-bar, scroll-wheel, and/orjog-dial type interfaces is especially problematic when implemented onrelatively small electronic devices. Additionally, for certain deviceswith very small display screens such as mobile phones and the like itmay not be feasible to implement any traditional scroll-bar, physicalscroll-wheel and/or jog-dial style interface due to display spacerequirements, even when such an interface would be useful to a user. Itwould also be desirable to provide the advanced scroll “acceleration”characteristics of traditional mechanical scroll-wheels and jog-dials toan electronic device in virtual (on display screen/user interface) form.The virtual scroll wheel of the current invention overcomes theselimitations while presenting a full function interface to a user.

Looking now to FIGS. 10 and 11, virtual scroll wheel 14 as depicted isgenerally rectangular in shape and is configured such that a user mayuse a finger, stylus, mouse, or other interactivity elementdirectionally up or down within the virtual scroll wheel 14 to effectmovement (i.e., scrolling, browsing, accelerating, etc.) of contentdisplayed in content display area 12. The actual shape, graphicalelements, and other “look and feel” characteristics of virtual scrollwheel 14 are the subject of design choice, however functionally,programming in or available to the device 10 may cause variousinteractions of an interactivity element with the area defined byvirtual scroll wheel 14 in display 1 to generate different behaviors andmovement of content in display area 12. For instance, as illustrated inFIG. 10, when a finger or stylus is caused to touch virtual scroll wheel14 at a substantially top portion and be dragged (and held) to a pointsubstantially half way down virtual scroll wheel 14, the contentdisplayed in content display area 12 will appear to scroll upward at adefined rate of speed (generally dependent on the distance between thefirst stylus screen touch location and the location at which the styluscame to rest on the screen, relative the area defined by virtual scrollwheel 14). Similarly, as illustrated in FIG. 11, when a stylus is causedto touch virtual scroll wheel 14 at a substantially top portion and bedragged (and held) to a point substantially at the bottom of virtualscroll wheel 14 (i.e., substantially twice the distance as shown in FIG.10), the content displayed in content display area 12 will appear toscroll upward at a rate of speed substantially twice that of the methoddescribed in relation to FIG. 10.

Other finger and stylus (and generally any user interactivity element)interactions are possible given the described method, for instance thestylus may be first caused to touch virtual scroll wheel 14 at asubstantially middle location, whereby a movement dragging the stylusdownward within virtual scroll wheel 14 causes increasingly acceleratedupward scrolling of the content in content display area 12, andsubsequent upward dragging of the stylus (generally without allowing thestylus to leave the surface of display 1) causes a gradual slowing, andthen reversal of direction in the scrolling of content. Likewise, forinteraction by a users finger in a manner similar to a scroll wheel on amouse, virtual scroll wheel 14 may be configured (through programming ondevice 10) to respond to successive swipes or drags (e.g., touches andreleases) of a users thumb in the direction of scroll bar orientation(as determined by programming in or available to device 10). Forinstance, downward vertical swipes or drags of a users thumb may causeupward scrolling of the content in content display area 12, while upwardvertical swipes or drags of a users thumb may cause downward scrollingof the content in content display area 12. It will be understood andappreciated that distances and directions traveled within virtual scrollwheel 14 for scrolling and acceleration purposes, and the particularmethod and technique of monitoring and calculating stylus screen touch,drag, pause, and off points are the subject of design choices that maybe dictated by, among other factors, device platform, operating system,programming language, etc., and are all well within the routine skill ofa programmer skilled in the art. By way of example, with reference toFIG. 11 b, in order to facilitate operation of the virtual scroll wheelvia a large (relative to overall screen size) blunt object such as afinger 18, the touch area responsive to input 16 may be larger than thedisplayed scroll wheel 14. Additionally, it will be appreciated that thesize, placement on a particular user interface or electronic device,shading, coloring, and other “look and feel” elements of the virtualscroll wheel of the current invention may vary widely without departingfrom the spirit and scope of the current invention. For instance, whenimplemented in conjunction with small electronic devices having touchscreens such as PDAs and the like, the defined screen area for operationof the virtual scroll wheel may consist solely of the rightmost edge (orany side of the screen) of pixels of the screen, without any visiblegraphic elements or indicators. As such, many methods of implementingand using the virtual scroll wheel of the current invention are possiblewithout departing from the spirit and scope of the current invention.

Scalable User Interface

For further maximizing available display area on a user interface andfor providing quick transition between a number of available interfacestates or parameters, or a continuum of interfaces states available to auser, a device 10 may utilize a scalable user interface 200. Inparticular, the scalable user interface 200 is designed to overcome bothsize and functionality restraints of portable electronic devices, andpresent an easily modifiable user interface consisting of two or moreinterface states to a user for operating and interacting with anapplication or program on device 10. In particular, to presently modifythe particular level of detail, resolution, or other user definablegraphic user interface characteristics, the user must use a number oftedious steps and/or operations on the device or software application toeffect the desired modifications. While methods such as skins,accessibility settings, and configurable toolbars have been used toallow a user to customize various aspects of a particular userinterface, such methods are laborious as indicated above, and lack theimmediacy of customization required for certain applications. As will beappreciated, for certain applications such as remote control userinterfaces for electronic devices, the lack of immediately customizableuser interfaces is problematic given that one main purpose of remotecontrol applications is to save users time and effort when interactingwith their home appliances and media content. As such, current userinterface customization methods are lacking for multiple userinterfaces, applications having multiple interface screens and elements,and for quickly switching between desired application interface statesand parameters. The scalable user interface of the current inventionovercomes these limitations while presenting an easily customizable,full function interface to a user.

Looking now to FIGS. 12-17, scalable user interface 200 as depicted isin a first or “default” state of customization (i.e., the defaultinterface for the particular software application). Particularly,scalable user interface 200 as illustrated is a content access interfacefor a remote control application, however scalable user interface may beany user interface and is not limited to any particular interface orsoftware application. Shown in FIG. 13 is second interface state 202,which generally consists of an enlargement of a portion of interface200, such that interface 202 may be more easily read under dim lightingconditions or by users having vision impairments, or in the case ofcontent selection, may be more easily interacted with via finger inputversus stylus input (e.g., for operation of device 10 similar to abutton based remote control or similar electronic device). Conversely,shown in FIG. 14 is third interface state 204, which generally consistsof a minimized portion of interface 200, such that interface 204 maydisplay more of the content set or other interface elements shown ininterface 200 such that a user can view and browse more availablecontent at a given time.

In order to effect modification of interface 200 to interface 202 or204, the user need not access any menus, settings pages, or the likeaccording to the current invention. Rather, programming on or accessibleto device 10 causes various interactions of an interactivity elementwith a predefined area of the screen to cause interface 200 to bereconfigured as interface 202 or 204. For example, as illustrated inFIG. 13, a user may operate a stylus, finger, mouse, or other userinteractivity element directionally from a point at substantially thetop center of the display to be dragged to a location substantially inthe upper left corner of the display in order to modify interface 200 tointerface 202. As will be appreciated, interfaces 200 and 202 may bediscrete interfaces (i.e. two individual interface states, such asbitmap or other graphical based interfaces) and interface 202 may notbecome active/loaded until the stylus has traveled a certain distanceacross the screen or reached a predefined screen location. Conversely,interfaces 200 and 202 may be two points on a continuum of possibleinterface states (i.e., such as would be possible with a zoomingalgorithm applied to interface 200, or the use of vector based interfaceelements). Likewise, as illustrated in FIG. 14, a user may operate astylus, finger, mouse, or other user interactivity element directionallyfrom a location at substantially the top center of the display to bedragged to a location substantially in the upper right corner of thedisplay in order to modify interface 200 to interface 204. As will beappreciated, interfaces 200 and 204 may be discrete or continuous asdescribed above.

While not to be taken as limiting, the exemplary method described aboveinvolves interacting with a top portion of the display in order tomodify one parameter (e.g., resolution, the enlarging or minimizing ofinterface elements) of interface 200. As shown in FIGS. 15-17, a sideportion of the display is shown as being used to modify a secondinterface parameter (e.g., detail, the adding or subtracting ofinterface elements) using a user interactivity element (finger, stylus,mouse, cursor, etc). Specifically, as shown in FIG. 16, interface state206, which includes control interface elements 210, is effected byoperating an interactivity element directionally from a location atsubstantially the top left of the display to be dragged to a locationsubstantially in the far most middle left portion of the display.Likewise, as shown in FIG. 17, interface state 208, which includescontrol interface elements 212 (which include original control interfaceelements 210 as well as an expanded detail view of the currently playingmedia) is effected by moving a user interactivity element directionallyfrom a location substantially in the far most middle left portion of thedisplay to be dragged to a location substantially in the bottom left ofthe display. As will be appreciated, interfaces 206 and 208 may bediscrete or continuous as described above. Additionally, for all ofstates 200, 202, 204, 206, and 208, (and any plurality of continuousinterface states between them according to one embodiment of the currentinvention) dragging of the stylus or other user interactivity element ina direction opposite those described above will effect a reversion tothe previous interface state, such that customization between allpossible states may be effected quickly by dragging the stylus or afinger across the top and left most edges of the display. As describedearlier in connection with FIG. 11 b, boundaries of the various touchscreen areas sensitive to input may be oversized as appropriate tofacilitate finger input. Furthermore, and in keeping with the teachingsand principles of the current invention, programming on or available tothe device 10 may enable additional or “hybrid” interface states whichconsist of a mixture of modified parameters one and two (i.e., interfaceresolution and interface detail) via dragging of the stylus or otherinteractivity element along the diagonal running between top and leftscreen edge portions (or any other selected areas) of the display. Inthis way, a larger number of discrete interface choices, or virtuallyunlimited choice of interfaces in the case of a continuum of interfaces,is possible by operating the stylus or other user interactivity elementalong a diagonal, or generally any multi-component (x, y) portion, ofthe display.

It will be understood and appreciated that the actual shape, graphicalelements, and other “look and feel” characteristics of interface states,the type and nature of modifiable interface parameters, as well aslocation and direction of defined display areas for interfacecustomization purposes, and the particular method and technique ofmonitoring and calculating stylus screen touch, drag, pause, and offpoints are the subject of design choices that may be dictated by, amongother factors, device platform, operating system, programming language,etc, and are all well within the routine skill of a programmer skilledin the art. As such, many methods of implementing and using the scalableuser interface of the current invention are possible without departingfrom the spirit and scope of the current invention.

Virtual Gesture Pad

For providing an enhanced user interface to a user such that usermovements and gestures on a touch screen display enable control andoperation of a software application, the device 10 utilizes a virtualgesture pad interface 300. In particular, the virtual gesture padinterface 300 is designed to overcome both size and functionalityrestraints of portable electronic devices, and present an easilycontrollable interface such that predefined movements of a stylus,finger, cursor, or other user interactivity element may controloperation of a software application, optimally without full attention ofthe user (i.e., single handed operation and/or operation while notviewing the display of device 10). For example, U.S. Pat. No. 6,396,523describes a remote control device having a touch pad element adapted toreceive gesture based input for initiation of various remote controlfunctions. In this manner, gesture input is converted to commandfunctions and control of the associated home entertainment equipment iseffected.

While the system described in U.S. Pat. No. 6,396,523 performsadequately when implemented on a larger scale device having bothphysical buttons and touch pad elements to facilitate interaction withinformation presented on a separate display device such as a televisionscreen, what is needed is a virtual gesture pad that functions toprovide advanced control based features quickly and interchangeably withtraditional soft button based controls for electronic devices havingintegrated touch sensitive screen displays. As will be appreciated, forcertain applications such as touch screen based remote control userinterfaces for electronic devices, the lack of easily accessible andintuitive gesture based user interfaces is problematic given that usersoften desire to operate the remote control application using a singlehand or finger, and/or without directly viewing the display screen. Assuch, current gesture based methods for control and operation ofsoftware applications are lacking for integrated touch screen devices.The virtual gesture pad of the current invention overcomes theselimitations while presenting an intuitive, interchangeable, gesturebased interface to a user.

Looking now to FIGS. 18-22, virtual gesture pad interface 300 asdepicted is in a “default” or non-gesture based input state (i.e. nogesture commands are accepted by the interface and associatedprogramming in the default state). In the default state shown in FIG.18, operation of and interaction with the software application iseffected by various soft buttons and/or other interface elements visibleon the display. Upon initiation, (which may consist of a button press,menu selection, initial gesture on the touch screen, voice command,system events, or any other user or device based input), a virtualgesture pad 302 may be overlaid on the screen display such that thebutton based commands of FIG. 18 are no longer available to the user.Shown in FIG. 19 is an exemplary virtual gesture pad 302, which in oneembodiment of the present invention, consists of a semi-transparentinterface overlay having gesture targets for effecting various softwarecontrols. For example, gesture target 310 consists of a single definedarea of the display (a substantially circular shape in FIG. 19) wherebya user touching anywhere within (or within a defined radius from) target310 will effect either a “play” or “pause” command successively witheach press. Gesture target 312 corresponds with target 314 in that theuser must touch the display substantially on or near target 312, thendrag a finger (or other interactivity element) across the screen totarget 314 before lifting the finger off the display in order to effecta “previous track” command from the associated software application.Likewise, gesture target 316 corresponds with target 318 in that theuser preferably touches the display substantially on or near target 316,then drags a finger (or other user interactivity element) across thescreen to target 318 before lifting the finger off the display in orderto effect a “next track” command from the associated softwareapplication. Notably, the “previous track” and “next track” commandsrequire gestures in substantially opposite directions with respect tothe display, thus enabling an increased level of error correction suchthat programming on or available to the device will be able toaccurately determine which command was intended by the user based on abrief user gesture. Gesture targets 320 and 322, and targets 324 and 326operate in a similar fashion in order to effect “next playlist” and“previous playlist” commands respectively. Targets 320, 322, 324, and326 are located in a different portion of the display from targets 312,314, 316, and 318, as well as being oriented at a different angle sothat further error detection and correction mechanisms will be possiblewith the virtual gesture pad.

FIGS. 20-22 illustrate another embodiment of the virtual gesture pad ofthe current invention. In particular, FIG. 20 illustrates asubstantially square central gesture target 330 which is surrounded byassociated command targets 332, 334, 336, 338, 340, 342, and 344.Central target 330 does not on its own correspond to a command orcontrol of the associated software, but rather functions as a centralpoint from which other commands can be initiated. For example, as shownin FIG. 21, a user touching the display in target 330, dragging a fingerdiagonally to target 332, and then removing the finger from the display(or pausing for a predetermined time period once within target 332) willeffect a “next artist” command through the associated software. Othercommands may be similarly effected by a user appropriately dragging afinger from central target 330 to the desired command target. Theconcept of pause detection and/or sensed directional change on the touchscreen (both of which are well known by those skilled in the art, aswell as well within the routine skill of a programmer skilled in art)can be used to accomplish the sending of multiple control or operationcommands in the same gesture. For example, as shown in FIG. 22, a usertouching the display in target 330, dragging a finger diagonally totarget 332, pausing for a predetermined time (or conversely causingprogramming in or available to device 10 to detect directional changesrather than pauses on the display) and then dragging her finger to (orpausing for a predetermined time period once within) target 334 willeffect a “next artist” command followed by a “next track” commandthrough the associated software.

It will be understood and appreciated by those skilled in the art thatvarious configurations of virtual gesture pads 302 and 304, includingdifferent shapes, sizes, and locations of the various gesture targetsand the particular method and technique of monitoring and calculatingfinger or stylus screen touch, drag, pause, and off points are thesubject of design choices that may be dictated by, among other factors,device platform, operating system, programming language, etc., and areall well within the routine skill of a programmer skilled in the art.For instance, using methods similar to those illustrated and describedin reference to FIGS. 20-22, virtual gesture pad type commands andoperation may be programmed and used on traditional hard button, ortouch panel based devices (i.e., the Kameleon brand remote control madeby Universal Electronics Inc. and sold under the One For All brandname). As will be appreciated, the touching and dragging of a usersfinger on such remotes may be somewhat more difficult than on LCD ortouch screen based devices, however the successive directionalactivation of various buttons and button combinations can effect gesturebased controls for such remotes using the inventive concepts of thepresent invention. Additionally, it may be advantageous for certainapplications or interface implementations to configure a virtual gesturepad such that gesture based functions and commands are possible withoutthe display of any associated gesture command screens, overlays,targets, or other display based representations. In this way a user mayinitiate gesture commands based on associated programming on the device,but retain full access and visibility of the associated applicationduring use. As such, many methods of implementing and using the virtualgesture pad interface of the current invention are possible withoutdeparting from the spirit and scope of the current invention.

Graphical Search Query Interface

For maximizing available display area on a user interface and forsimplifying the process of indicating to a user the results of a searchquery and allowing the user to interact with the search query results, adevice 10 may utilize a graphical search query interface 400 as shown inFIGS. 23-25. In particular, the graphical search query interface 400 isdesigned to overcome both size and functionality restraints of portableelectronic devices, and present a graphical representation andinteractive interface to a user for browsing through one or more searchquery results (i.e., any search query performed by a user for quicklylocating desired text, data, photo, music, video, program guideinformation, and/or other related content) and conveniently displayingsuch results within the available display area of a device. Thegraphical search query interface of the current invention overcomes thelimitations in the prior art and presents a full function search queryinterface to a user.

Looking now to FIGS. 23-26, graphical search query interface 400 asdepicted consists of at least two query interface areas 402 and 404 forindicating to a user the relation of a search query criterion 402 (i.e.,musical albums beginning with the letter “S”) with returned data orresults set 404. In one exemplary configuration, query interface area402 and results set 404 are generally circular in shape, and are movedsuch that they overlap one another as results set 404 begins to matchitems with query criterion 402 (i.e., as programming on device 10performs a search based algorithm designed to match a given text basedquery with data, files, or other content on or available to device 10).For example, as shown in FIG. 24, matching results set 406 occupies thearea created by the overlap of query interface area 402 and results set404. In this way an indicator of the matching query results, and therelative success of the user's particular search terms in matching anassociated data set, is presented in an appealing, intuitive manner.Optionally, as shown in FIG. 25, a second, or sub-query criterion may beentered by a user, and is represented graphically as sub-query area 408.Also shown in FIG. 25 is results set area 412, which represents onlythose results returned by the search query represented by query area402, and results set area 414, which represents only those resultsreturned by the search query represented by sub-query area 408. Resultsset area 410 represents the matching results returned by the combinationof both criteria, i.e., the search query and sub-query. As will beappreciated, the search query, and sub-query will return individualresults sets pertaining to the particular terms searched, and in somecases there may be some commonality between items in the respectiveresults sets (e.g., in the case where a user searches for terms orproperties that pertain to common data fields for a particular data orcontent set). For example, the search query relating to query area 402could be a content search for musical albums beginning with the letter“S,” while the sub-query relating to sub-query area 408 could be acontent search for musical albums released in the year “1985.” A usermight have music files on a device, such as a computer, such that someresults (i.e., those represented by results area 412) are all musicalalbums beginning with the letter “S” which were not released in 1985,some results (i.e., those represented by results area 414) are allmusical albums released in the year “1985” which do not begin with theletter “S,” and some results (i.e., those represented by results area410) are all musical albums beginning with the letter “S” which werealso released in the year “1985.”

Given the above disclosure and associated figures, it will beappreciated by those skilled in the art that the described results setconstitute Venn diagrams as applied to Boolean search methods, howeverit is one object of the present invention to present these results setsto a user in a dynamic and interactive fashion. As shown in FIG. 26,having entered a particular search query and optional sub-query, a usermay dynamically modify the displayed results set (i.e., the actualcontent reference data displayed in the display area 12 of device 10, orat any other location of display 1) by interacting with one or more ofthe graphically displayed results set areas 406, 410, 412, and 414and/or query areas 402 and 408 of graphical search query interface 400using a finger, stylus, or any other user interface element. In theabove described example, by simply touching or selecting result set area410, all data referencing all musical albums beginning with the letter“S” which were also released in the year “1985” will be displayed in thecontent display area of the device for further user selection orinteraction. The user may interact with results set areas 406, 412, and414 in a similar fashion. As shown in FIG. 26, a user may also touch anddrag one or more of the query areas 402 and 408, and result set 404 inorder to modify one or more search query parameters, also causingdynamic modification to the displayed results set shown in the displayarea of device 10. For example, dragging sub-query area 408 to the rightor the left in the present example may cause the search term itself tobe dynamically modified (e.g., scroll forward though the years field assub-query area is dragged to the right with respect to its originalposition, and scroll backward through the years field as sub-query areais dragged to the left with respect to its original position). As such,graphical search query interface 400 and the associated displayedresults sets are configured (via programming in or accessible to device10) to dynamically change depending on the particular search terms, aswell as user interactions with graphical search query interface 400.

It will be understood and appreciated that the size, placement on aparticular user interface or electronic device, shading, coloring, andother “look and feel” elements of the graphical search query interfaceof the current invention may vary widely without departing from thespirit and scope of the current invention. Additionally, the particularmethods and techniques for generating and allowing interaction with thegraphical search query interface will be apparent from the descriptionsherein, as well as well within the routine skill of a programmer skilledin the art. For instance, various algorithms for causing search terms orresults sets to be modified can be implemented in conjunction with thegraphical search query interface of the current invention to accomplishdesired user experience goals for a specific device or software productwithout departing from the spirit and scope of the present invention.

Graphical Indication of Device Mode and State

Device 10 and associated application software may be configured tooperate networked control environments wherein some or all of the homeappliances and components are configured to be network enabled andinteroperable, and which include one or more media server devices andmedia rendering devices (e.g., as defined and described in the UPnPand/or HAVi specifications which can be found at http://www.upnp.org andhttp://www.havi.org respectively and which are incorporated herein byreference in their entirety). Turning now to FIG. 27, when the exemplaryuser interface methods are applied to the control, selection, anddelivery of media content from server devices to rendering devices, thedisplayed user interface 200 may include areas 500, 502 used to conveyinformation to the user regarding the currently selected server orrendering device and the types of content available from it or supported(for playback or other processing) by it. This is especiallyadvantageous when the interface is applied to a networked consumerentertainment environment wherein multiple server and rendering devicesexist, each with varying media based services and capabilities. In theillustrative example of FIG. 27, area 502 displays the name of thecontent server currently selected, and area 500 displays arepresentation of the media server currently selected. Icon 504 may berepresentative of a server device such as a computer, networked storagedevice, a media server device, etc. In one instance, area 500 mayprovide additional information regarding the types of content availablefrom that server. With reference to the detailed views 500.1 thru 500.6,an icon 504 representative of the selected server is accompanied bycontent type sub-icons 506, 508, 510 indicating the availability ofmusic, photographs (i.e., still pictures) and video contentrespectively. By way of example, 500.2 illustrates a representation of aserver from which both music and photographic content is available,while 500.3 illustrates the representation of a server from which onlymusic is available. Additionally, the appearance of one of the contenttype sub-icons may be altered (highlighted, flashing, animated,inverted, etc.) to further indicate to the user which type of content iscurrently listed and available for selection in the content display area12. In this context, by way of detailed example a label such as“Chicago” in area 12 may ambiguously reference either a movie, a rockband, or a simple photograph of the Lake Michigan waterfront. Alterationof the appropriate icon 506′ as illustrated in 500.4 or 500.5 (Chicagothe band) or 510′ as illustrated in 500.6 (Chicago the movie) may beused to remove this ambiguity.

It will be appreciated that in the case of servers offering multiplecontent types, switching between content types to be listed in area 12may be accomplished by touching one of the sub-icons 506, 508, 510 witha stylus, finger, cursor, etc. Such interaction may occur directlywithin the area 500 illustrated, or alternatively an initial touch maycause an enlarged semi-transparent version of this area to overlay thedisplay in a manner similar to that described earlier in connection withFIG. 9.

In another aspect of the current invention, area 500 may provideinformation regarding a rendering device currently selected rather thana content server. In this case, icon 504 may be representative of arenderer such as a stereo amplifier, TV set, Smart Display, LinksysDigital Media Adapter, HP Digital Media Renderer, or other hardware orsoftware based media renderer, and sub-icons 506, 508, 510 may indicatewhich types of content that rendering device is capable of processing.In a manner similar to that described above, one of the sub-icons may behighlighted to indicate the currently selected, or currently playingmedia type. Additionally, in order to indicate to a user that therendering device represented by icon 504 is one of a plurality ofavailable rendering devices for use, an alphanumeric indicator 505 maybe associated with icon 504. Alphanumeric indicator 505 may representthe number of rendering devices available, or may represent the order ofthe particular rendering device being represented by icon 504 from anumber of available rendering devices. Alphanumeric indicator 505 may beplaced anywhere on or near icon 504 such that a user may associateindicator 505 with icon 504, and indicator 505 may be configured torespond to touch by a stylus, finger, cursor, or other interactivityelement in order to switch to a different rendering device.

For situations where a selected renderer does not include certain mediarendering functions, the icons 506, 508, or 510 may include anadditional availability indicator 512 (i.e., an “X” placed next to theicon representing the unavailable media function as shown in FIG. 28) toquickly indicate to a user that a certain media function or functionsare not available for the selected renderer. User interaction with thearea 500, or on icon 504 may be configured to bring up a list ofavailable renders for selection represented in FIG. 29 by icons 504.1,504.2, 504.3, and 504.4. Associated with each media renderer icon arevarious sub-icons 506, 508, and 510 as described above, in this caseshown being configured such that only icons representing the availablefunctions of each renderer are displayed. An additional status indicatoricon 514 may be used in conjunction with each sub-icon 506, 508, and 510to indicate the current status of each rendering function of theassociated rendering device. For example, icon 514 may represent thatthe function associated with the icon directly adjacent icon 514 (i.e.,sub-icon 508 which represents photo playback functionality in theexample shown in FIG. 29) is currently active. In this way a user may bepresented a compact visual indication of all available media renderingdevices, their associated available functions, and the status of eachfunction.

Similarly, as shown in FIG. 30, user interaction with area 502 may beconfigured to bring up a list of available content servers 502.1, 502.2,and 502.3 for status indication and content selection. Status indicatoricon 514′ may be used to indicate that the server directly adjacent icon514′ (i.e., server 502.1 in the example shown in FIG. 30) is currentlyactive. Although not shown in FIG. 30, additional sub-icon similar toicons 506, 508, and 510 may be used in conjunction with the display ofavailable content servers to indicate the types of content available oneach content server. When used in conjunction with area 500, anindication of the renderer (icon 504) being used for the playback ofmedia from the selected content server 502.1 may be displayed, as wellas the status of content playback via sub-icons (i.e. icon 506′).

It will be appreciated that these concepts may be extended acrossmultiple areas of the display surface in order to simultaneously presentthe status of both a server and a rendering device. For example, thespace shown occupied by the setup “wrench” icon 520 could in certaincircumstances be replaced by a display element similar to area 500,allowing simultaneous presentation of the capabilities of both a serverand a rendering device using similar visual elements. In this instanceit will be understood that in addition to the touch inputs previouslydescribed, switching and routing of media streams, and rendering devicesmay also be accomplished by dragging a stylus, finger, cursor, etc.,from one of the sub-icons of a server representation into a rendererrepresentation, and vice-versa. It will be further understood andappreciated that the size, placement on a particular user interface orelectronic device, shading, coloring, and other “look and feel” elementsof the described icons and indicators of the current invention may varywidely without departing from the spirit and scope of the currentinvention.

The system and process of the present invention has been described abovevia the use of illustrative graphical user interface elements anddesigns. It is understood that unless otherwise stated to the contraryherein, the functions and methods by which these are generated andrendered may be integrated in a single physical device or a softwaremodule in a software product, or may be implemented in separate physicaldevices or software modules, without departing from the scope and spiritof the present invention.

It is to be appreciated that detailed discussion of the actualimplementation of each graphical display element and user interfacemethod is not necessary for an enabling understanding of the invention.The actual implementation is well within the routine skill of aprogrammer and system engineer, given the disclosure herein of thesystem attributes, functionality, and inter-relationship of the variouselements in the system. A person skilled in the art, applying ordinaryskill can practice the present invention without undue experimentation.

While the invention has been described with respect to variousillustrative examples, it will be apparent to those skilled in the artthat various modifications and improvements may be made withoutdeparting from the scope and spirit of the invention. Accordingly, it isto be understood that the invention is not to be limited by thesespecifically illustrated examples.

All of the cited references are incorporated herein by reference intheir entirety

1. A portable electronic device adapted to accept a gesture based inputfrom a user, comprising a touch screen pad used to accept the gesturebased input, the touch screen pad having a plurality of gestureaccepting areas, each gesture accepting area being arranged to accept agesture in a direction that is unique with respect to the other gestureaccepting areas.
 2. The portable electronic device as recited in claim1, further comprising a transmission circuit for transmitting commandsto one or more electronic devices in response to the gesture basedinput.
 3. The portable electronic device as recited in claim 1, whereineach gesture accepting area is positioned on the touch screen display ata location that is unique with respect to the other gesture acceptingareas.
 4. The portable electronic device as recited in claim 1, whereinall of the plurality of gesture accepting areas are initiated on thetouch screen display in response to an initial gesture being provided tothe touch screen pad.
 5. A portable electronic device adapted to accepta gesture based input from a user, comprising a touch screen pad beingused to display content and to accept the gesture based input, whereinthe gesture based input comprises a first characteristic and programmingassociated with the touch screen pad functions to modify a firstparameter of the content being displayed as a function of the firstcharacteristic.
 6. The portable electronic device as recited in claim 5,wherein the first parameter of the content comprises a zoom factor ofthe content being displayed on the display.
 7. The portable electronicdevice as recited in claim 6, wherein the zoom factor of the contentbeing displayed is modified from a first discrete zoom level to a seconddiscrete zoom level as a function of the first characteristic.
 8. Theportable electronic device as recited in claim 6, wherein the zoomfactor of the content is modified on a continuum from a first zoom levelto a second zoom level as a function of the first characteristic.
 9. Theportable electronic device as recited in claim 6, wherein the firstcharacteristic comprises a distance defined by the gesture based input.10. The portable electronic device as recited in claim 6, wherein thefirst characteristic comprises a vector defined by the gesture basedinput.
 11. The portable electronic device as recited in claim 5, whereinthe programming associated with the touch screen pad is initiated inresponse to a gesture based input first being provided to the touchscreen pad.
 12. The portable electronic device as recited in claim 5,wherein the first parameter comprises a selection of graphical userinterface input elements displayed as a part of the content.
 13. Theportable electronic device as recited in claim 5, wherein the firstcharacteristic comprises an end point of the gesture based input on thetouch screen pad.
 14. The portable electronic device as recited in claim5, wherein the first characteristic comprises a distance defined by thegesture based input.
 15. The portable electronic device as recited inclaim 5, wherein the first characteristic comprises a vector defined bythe gesture based input.
 16. The portable electronic device as recitedin claim 5, wherein the first characteristic comprises a number of timesthe gesture based input has been repeated in succession.
 17. Theportable electronic device as recited in claim 5, wherein gesture basedinput comprises a second characteristic and the programming associatedwith the touch screen pad functions to modify a second parameter of thecontent being displayed as a function of the second characteristic. 18.The portable electronic device as recited in claim 17, wherein the firstparameter of the content comprises a zoom factor of the content beingdisplayed on the display and the second parameter comprises a selectionof graphical user interface input elements displayed as a part of thecontent.
 19. The portable electronic device as recited in claim 17,wherein the first characteristic comprises a value provided by thegesture based input with reference to an x-axis of the touch screen padand the second characteristic comprises a value provided by the gesturebased input with reference to a y-axis of the touch screen pad.